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AESTRACT 

This presentation clarifies the role of evaluation 
in public schools and discusses major problems which block the 
effectiveness of the role. Evaluation is used in schools to produce 
information that helps decision makers choose among alternative 
courses of action. Purpose, rather than methodology, distinguishes 
evaluation from research. The need for care, precision, and logical 
thought, however, is common to both. One set of major problems is 
technical and involves such things as measurement and choice of 
appropriate analysis techniques. A second set revolves around the 
need for decision makers to follow a rational approach and to avoid 
reverting to "gut" feelings in every stress situation. Finally, 
evaluators must produce pertinent information, in the most useful 
format, at key points in time. (Author) 
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It’s a real pleasure to be with you here today, and I wane to 
commend all of you for remaining throughout this lengthy cession. 

That demonstrates real interest, plus a good deal of staying power. 

Appearing with this group of college professors makes me feel somewhat 
like Adolph Rupp when he made a speech one time in Indiana. He’s the 
basketball coach at the University of Kentucky, as you may knot?. When 
Rupp got up to begin his remarks, he said that the title of his presen- 
tation was, "A Stranger in the Promised Land." You’ll see on the program 
that my topic is supposed to be "Research in Public Schools: Purpose and 
Problem Areas.” But after hearing the previous papers, I’ve decided to 
change the title to "A Stranger in the Promised Land." 

We’ve heard a good bit of talk this morning about the difference 
between "research" and "evaluation”. In terms of what I’ve heard here 
and read previously, I hate to describe the kinds of activities in which 
I am engaged, because I’m sure that I will be placed in that somehow 
less-than-holy category of "evaluator". My feeling is that, so far as 
public school practice is concerned, the distinction is really rather 
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meaningless. It is a sort of semantic argument. The important thing is 
what one is doing, and not whether he considers himself engaged in research 
or in evaluation or in some other kind of activity. 
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I had planned to say something about the purpose of "research” 

or "evaluation” in public schools, but in doing so I’m going to be 

repeating what has already been discussed here in some detail. As 
• # * 

several of our speakers have noted, evaluation is a set of activities 
conducted in support of the decision process. Research and evaluation 
exist at the public school level as a service function. It is that, purely 
and simply, and there is no other reason for its being. There is no 
research . function of intrinsic value j research is of use only to. the 
extent that it can contribute to the primary purpose of educating children. 

All of this is easy enough to say and to conceptualize in the 
abstract. But as those of us who are attempting to do something in the 
public school setting have learned, actually making research and evaluation 
play a more central and decisive role in school operation is a difficult 
task. We talk rather glibly about producing information, but it is 
extremely difficult to be precise in defining what information is needed 
in specific cases, and producing it with at least sufficient validity 
that the product is worth the effort and expense. 

The information in each case depends on who the decision maker 
is, the decision he faces, the areas of uncertainty he needs resolved, and 
the like. A well-worn say of saying this is that the information needed 
depends on the questions to be answered. If one can identify the important 
questions, then he can proceed to determine the information required. 

A useful way to approach problem definition in school research, 

I believe, is to borrow one of the guiding tules used in journalism. I 
used to be a writer myself , but nobody wanted to read my stuff , so I 
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had to come into the research business. Journalists speak of four W’s 
and an H. I* 11 drop one W and say that to define a problem, that is to 
identify important questions, the school researcher needs to concentrate 
on three W*s and an H as follows. 

The first W is Who . Who is the decision maker to be served? This 
may be anyone, from the Superintendent of Schools to a classroom teacher. 

It may be a personnel specialist, the curriculum director, or someone 
concerned- with facilities. It can be any professional level decision 
maker in the public school setting, and the information needs of the person 
will obviously depend in several respects on the nature of his tasks. One 
has to approach and deal with different * people in different ways. Super- 
intendents, for example, tend to be prima donnas, and a certain style of 
working with them is necessary. The school researcher must leant something 
about the particular decision maker and the context of his operation before 
the other questions stated below can be adequately answered. 

The next W is What . What is the nature of the decisions that the 
person faces, and which specific ones will the research and evaluation 
specialist try to help him make? The decision maker may have trouble 
determining which questions are most important. In fact,- this is almost 
always a troublesome point, and a useful way to approach it is to try to 
zero in on areas of uncertainty. What are the crucial points on which the 
decision maker is uncertain, and what kinds of information are needed to reduce 
the uncertainty? Another way of saying this is," what kinds of information 
will increase the known probabilities . that certain outcomes will result from 






certain actions? 
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The third W is When . When does the decision maker need the 
information? That is, at what points in time do the decisions have to 
be made, and how much lead time is needed with the information in hand. 

Timing is vital to the public school researcher, and it is essential to 
his chances of efficacy. It is the issue on which a good proportion of 
public school research and evaluation effort to date has been rendered . 
impotent. The production of information has been out of phase timewise 
far too often. It has almost been, in a great many instances, as if . 
information was produced independently of. any decision scheduled. I f m 
a real expert on this point, because I’ve managed to fail in this way so 
many times. Many of us have produced beautiful documents which amazed 
and made envious our fellow researchers, but which unfortunately were not 
available when the relevant decisions had to be made. As a very common 
example, we produce beautiful program evaluation documents in the summer. 

The only problem is that decisions about the program for the next year 
were made last May. In school research and evaluation work a schedule 
of decisions must be worked out, and the production of information must 
be tied to that schedule. Timing is essential. 

The H in our three W’s and an H is How . How should the information 
be presented? In addition to making sure that he is producing the information 
really needed at the time it is needed, the school researcher must be 
‘ concerned with getting it to the decision maker in the format most useful 
to him. Those beautiful reports referred to a moment ago often run to 
50 or 60 pages, and they’re full of esoteric terms and fancy statistics 
which thrill the hearts of other researchers, but which have no meaning 
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to ordinary human beings. Reporting must be built around the decision 
maker 1 s needs. If he needs a one-page summary in simple language, this 
is what he should get. If he needs five pages which consist mostly of 
graphics, this is what he should get. It is the decision maker, and not 
the researcher, whose needs are being met in reporting. If the school 
researcher wants to communicate with his peers , contribute to the 
scholarly literature, and the like, this is highly commendable. He should 
do it as time permits. But it .is a separate issue from the reporting 
which is an integral part of his function as a support specialist to 
decision makers. 

The question of determining which decision maker is to be served 
is a difficult one. Does a school researcher or evaluator assume an 
entirely passive posture and react only to requests for infoimation delivered 
to him? If he does, he is in real danger of dealing mostly with relatively 
trivial issues. Does he concentrate in a few areas, such as instructional 
program evaluation, and leave other important ones, such as finance, 
alone? His resources and his expertise are both likely to be limited, 
and he cannot be all things to all people. 

Research staffs' in school systems normally are physically located 
in the central administration building, and the majority of their contact 
is usually with central administrative staff. This in itself makes them 
highly suspect to people out in the schools. Wii-hin the central admin 
istrative staff, the research and evaluation Specialist has the problem of ♦. 
relating to those who hold the key power positions and whose decisions , 
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therefore, really affect what happens in the system. On top of this, we 
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all know that the most important personnel for affecting real change may 
not be in the central office at all. They may be in the schools. 

Perhaps most of the effort should be concentrated at the local 
school level. Impact in public education finally boils down to the 
interaction between the teacher and the pupil. If change doesn't occur, 
at that level, then nothing else really matters. This does not necessarily 
.mean that the researcher must always be working directly with teachers, 
however. Development of a good reading program -with a- small but well- 
organized group might have a much greater effect on pupil- teacher interaction 
than working directly with a hundred separate teachers. The school 
researcher faces a dilemma in this respect, therefore. To what extent 
does he react to the needs expressed by field personnel? How is the right 
decision maker identified to make this kind of determination? 

We f ve heard some talk about general izability this morning and about 
evaluation being an area where generalizability is not a concern. I 
disagree with this point of view. I think that generalizability must be 

a concern in conducting the research-like activities in schools. The 

* 

.school research and evaluation specialist could spend all of his time 
working on specific problems in individual schools and perhaps help a 
handful of teachers. I could very easily have 500 people engaged in this 
kind of activity in a system the size of Philadelphia. The cost effectiveness 
of this operation would certainly be questionable, however. Can n school 
district afford to have relatively high salaried research specialists 
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working on specific problems which have no gen€*ralizability, at least 

in the context of the local district? The researcher must focus on the 

pupil-teacher interaction, but he must also affect change throughout the 
• # " 
system to justify the expense of maintaining him. 

As already noted, much of the discussion to this point seems to 
assume something of a passive posture on the school researcher’s part, 
and, while he cannot avoid investing part of his energies in servicing 
requests,- he must find some way to be more intrusive if he is to play a 
really important and central role in school operation. Few of us at this 
stage know how this is to be accomplished. Reporting directly to the 
superintendent and getting a clear commitment from him and other top decision 
makers to the research and evaluation process is undoubtedly important. 

At the same time, a working relationship with the field, perhaps through 
advisory committees, is also necessary. 

A big point was made earlier in this paper about the purpose of 
research and evaluation in the public school setting. It is purpose, I 
think, that differentiates this type of work from the activities of 
university researchers and others who are in some way engaged in the 
research and evaluation enterprise in education. I would question Mr. 

Stuff lebeam on a point he made in this regard. He presented what I 
would say is something of a false dichotomy. That is, he spoke of 
experimental research and evaluation activities as two mutually exclusive 
categories. I would rather name the categories "evaluation" and "research”, 
and I would argue that experimentation can be used in either setting. 
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I believe, in other words, that methodology is not a distinguishing 
characteristic between the research and evaluation roles. It seems to 
me that, as Stake and Denny have noted, the basic inquiry paradigm is the same 
whether one is engaged in basic research or in evaluation. And I believe 
that the same set of methodologies must be chosen from. The classical . 
experimental design will be workable much more often in the laboratory, 
of course, and the proportion that this and other methodologies are used 
will differ in the different settings. It is the proportion of use and 
not the available methodologies, however, which differs. Both researcher 
and evaluator have the same set of methodologies from which to chose. 

I am personally convinced that different models, such as the CIPP 
model of Stuf flebeam, Provus' discrepancy model, Stake’s judgumental model, 
the anthropological model that Ianni is developing at Columbia — these 
different models and different methodologies produce different kinds of 
information under different sets of circumstances. And I think any one 
of them should be used when appropriate. Therefore, I believe that the 
school research and evaluation specialist must be flexible, knowledgeable 
about many different approaches and the related statistical techniques, 
and capable of adjusting his activities to specific situations. 

All of this puts a premium back on problem definition. One must 
carefully determine the three W’s and the H discussed previously and, 
based on this, must state clearly the questions to be answered. He must 
then examine the context in which he will work and determine such things .■ 
as the amount of control which can be exerted, and the availability of 
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different kinds of data. With this kind of background, he can then 
select the appropriate methodologies. 
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I’ve been talking this morning about problems a school research 
and evaluation specialist faces in trying to impact the decision process. 

To be completely honest, I would have to say that we have not had a great 
deal of success in affecting public school practice to this point in time, 
and a good part of the problem is ours. Many improvements in our practice 
are obviously needed. 

Even if we manage to work to perfection, however, we can still be. 
impotent, because there’s another side to the coin. The other side is 
the decision makers and their behavior, and this is a side which seems to 
me to be ignored a greal deal in discussions of research and evaluation in' 
education. None of the other speakers this- morning, for example, has said 
anything about how one gets decision makers to use information that is 
delivered to them. It seems evident to me that decision makers, in addition 
to giving lip service to the need for research and for more informed 
behavior on their part, need to learn how to help define information needs 
and to use the information when it is delivered. School superintendents 
and others in high administrative positions have strong survival instincts, 
and few of them got where they are by making use of research-based 
information. They have habits, reinforced by years of practice, of reacting 
to crisis situations in terms of their own gut-level feelings and the 
political realities they perceive. This kind of behavior has worked for 
them before, and they tend to revert to it when the chips are down. 

Change in their behavior is needed, and producing this change . is part of the 



task we face. 
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Research utilization at the public school level is an area which 
has not been studied very carefully, but. which must be addressed in depth, 
I am firmly convinced, before we can succeed in making the research and 

•4 

evaluation process impact decision making in public schools. 

I appreciate very much the attention you've given me, and I have 
enjoyed being a member of this panel. I hope very much that we can do 
something to improve research and evaluation practice at the public school 
level. Thank you. 
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